Help for downloading Youtube video "server replied: Forbidden"

[rsilva]

I'm trying to download a video from youtube.

I'm using something like in this python code:
https://github.com/rg3/youtube-dl/blob/master/youtube-dl

But rewrote it for my code.
Everything works fine, to get the real download link.

I put this link: http://www.youtube.com/watch?v=AGAIq5_sb4o&feature=related

And get this: http://v11.lscache8.c.youtube.com/videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2C itag%2Calgorithm%2Cburst%2Cfactor&fexp=912503&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=B0C65F8BAD05232C6DDA4691E2AD6A1FA73E844E .01F3D85030E8977C6581CC1EA7E302DB2E45F323&expire=1305608400&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a

At here, everything ok, now I'm trying to download the video using this link.
But I got an error in the reply, this error:

"Error downloading http://v11.lscache8.c.youtube.com/videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2C itag%2Calgorithm%2Cburst%2Cfactor&fexp=905236%2C912503&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=B0C65F8BAD05232C6DDA4691E2AD6A1FA73E844E .01F3D85030E8977C6581CC1EA7E302DB2E45F323&expire=1305608400&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a - server replied: Forbidden"

This is my code:



void Downloader::start()
{
YtInfo* info = qobject_cast<YtInfo*>((QObject*)m_item->data(Qt::UserRole + 1).toInt());

QNetworkRequest request = QNetworkRequest(QUrl(info->getHigherQualityDownload()));
//request.setRawHeader("User-Agent", "Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10");
request.setRawHeader("Youtubedl-no-compression", "True");

//m_reply = m_manager->get(request);
m_reply = m_manager->head(request);

connect(m_reply, SIGNAL(downloadProgress(qint64,qint64)), SLOT(downloadProgress(qint64,qint64)));
connect(m_reply, SIGNAL(finished()), SLOT(finished()));
}

void Downloader::finished()
{
qDebug() << m_reply->errorString();
m_reply->deleteLater();
}


info->getHigherQualityDownload()
Returns the best quality video avaliable for download.
The link that I've posted.

I've tried changing the User-Agent header, but doesn't work too.

The video link works in the browsers but not inside my Qt Application.

I've tested it in the Qt Browser Demo and the link doesn't work too.
Just got an "non-exiting" page.

But works in Opera, Mozilla and Chrome. (I've not tested it on IE).

The download works even if I'm logged in a youtube account or no.

Someone can help me ?

[wysota]

So what do you expect us to do? Take a network sniffer and compare the traffic from your app with the one from a full network browser.

[rsilva]

I'll try it using wireshark.
I've asked because I've thought that someone may have this problem too.
Or someone knows how to solve it, or may be someone knows if this needs header-tricks or something else.
Or a way to find why this is happening, but thanks for answering.

[wysota]

Look, how would anyone know what is wrong with your code if the whole logic is inside this YtInfo class which we don't have any info about? In general your problem is not Qt related but YouTube related -- you don't know how to download a file from YouTube and not how to do it using Qt. If you were to do it using plain native sockets, you'd have exactly the same problem. By the way the "real" link you pasted is invalid so there is no way of us checking whether anything works.

[rsilva]

It's valid here. I can use it for download here.
Yeah, I think it's youtube related.

I know how to do this manually, just get this link and use it for download, it's working here.
The only problem is, download this real link with Qt, that doesn't work even with Qt Browser Demo.

In Opera request for the video I have:



Request Method: GET

Request URI: /videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2C itag%2Calgorithm%2Cburst%2Cfactor&fexp=912601&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=40EA572A3CC8F901C65EB029E3F595C090C3F85F .06EDF73184B87C4158D7C0A430408969F43FDC2E&expire=1305612000&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a

Request Version: HTTP/1.1

User-Agent: Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10\r\n

Host: v11.lscache8.c.youtube.com\r\n

Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1\r\n

Accept-Language: pt-BR,pt;q=0.9,en;q=0.8\r\n

Accept-Encoding: gzip, deflate\r\n

Connection: Keep-Alive\r\n

\r\n


QNetworkAccessManager request:



Request Method: HEAD

Request URI: /videoplayback?sparams=id%252Cexpire%252Cip%252Cipb its%252Citag%252Calgorithm%252Cburst%252Cfactor&fexp=909406%252C912503&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=40EA572A3CC8F901C65EB029E3F595C090C3F85F .06EDF73184B87C4158D7C0A430408969F43FDC2E&expire=1305612000&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a

Request Version: HTTP/1.1

Youtubedl-no-compression: True\r\n

Connection: Keep-Alive\r\n

Accept-Encoding: gzip\r\n

Accept-Language: pt-BR,en,*\r\n

User-Agent: Mozilla/5.0\r\n

Host: v11.lscache8.c.youtube.com\r\n

\r\n


And now, I've just changed my code to this:



void Downloader::start()
{
YtInfo* info = qobject_cast<YtInfo*>((QObject*)m_item->data(Qt::UserRole + 1).toInt());

QNetworkRequest request = QNetworkRequest(QUrl(info->getHigherQualityDownload()));
request.setRawHeader("Accept", "text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1");
request.setRawHeader("Accept-Encoding", "gzip, deflate");
request.setRawHeader("Accept-Language", "pt-BR,pt;q=0.9,en;q=0.8");
request.setRawHeader("User-Agent", "Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10");

m_reply = m_manager->get(request);

connect(m_reply, SIGNAL(downloadProgress(qint64,qint64)), SLOT(downloadProgress(qint64,qint64)));
connect(m_reply, SIGNAL(finished()), SLOT(finished()));
}


And I get the same "server replied: Forbidden" again.

This is the request now:



Request Method: GET

Request URI: /videoplayback?sparams=id%252Cexpire%252Cip%252Cipb its%252Citag%252Calgorithm%252Cburst%252Cfactor&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=96B89386D0B3947AA32331AD917B7F23AD71AB2C .C8B734A2140BF665BF9A172C45126A38BCD1CCF9&expire=1305622800&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a

Request Version: HTTP/1.1

Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1\r\n

Accept-Encoding: gzip, deflate\r\n

Accept-Language: pt-BR,pt;q=0.9,en;q=0.8\r\n

User-Agent: Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10\r\n

Connection: Keep-Alive\r\n

Host: v11.lscache8.c.youtube.com\r\n

\r\n


With the same request, I'm getting that forbidden error.
What I can do now ? The request is exactly the same.

I'm sending the YtInfo class (ytinfo.h and ytinfo.cpp) in the Attachments.

The main functions is:
That is called with this link:



QString YtInfo::m_videoInfoUrl = "http://www.youtube.com/get_video_info?&video_id=%1%2&ps=default&eurl=&gl=US&hl=en";

//%1 = video id
//%2 = "&el" type




void YtInfo::_loadInfo()
{
if (m_el < 0 || m_el > 4) return;

QList<QString> types;

types.append("");
types.append("&el=detailpage");
types.append("&el=embedded");
types.append("&el=vevo");

QString infoUrl = m_videoInfoUrl.arg(m_id, types.at(m_el));
QNetworkRequest request = QNetworkRequest(QUrl(infoUrl));

m_manager.get(request);
}

void YtInfo::_infoLoaded(QNetworkReply* reply)
{
if (reply->error())
{
emit error(this, reply->errorString());
}
else
{
QUrl url("http://.../?" + reply->readAll());

if (url.hasQueryItem("fmt_url_map"))
{
QString fmtmap = url.queryItemValue("fmt_url_map");

if (!fmtmap.isEmpty())
{
fmtmap = QUrl::fromPercentEncoding(fmtmap.toAscii());
QStringList urlmap = fmtmap.split(',');

foreach (QString fmturl, urlmap)
{
QStringList fmt = fmturl.split('|');
m_avaliable[fmt.at(0)] = fmt.at(1);
}

if (url.hasQueryItem("author"))
{
m_author = url.queryItemValue("author");
m_author = QUrl::fromPercentEncoding(m_author.toAscii());
}

if (url.hasQueryItem("title"))
{
m_title = url.queryItemValue("title");
m_title = QUrl::fromPercentEncoding(m_title.replace('+', ' ').toAscii());
}

if (url.hasQueryItem("length_seconds"))
{
QString tmp = url.queryItemValue("length_seconds");
tmp = QUrl::fromPercentEncoding(tmp.toAscii());

m_length = tmp.toInt();
}

emit ready(this);
}
else
{
emit error(this, "Empty fmt_url_map.");
}
}
else
{
if (url.hasQueryItem("reason"))
{
if (url.hasQueryItem("errorcode"))
{
if (url.queryItemValue("errorcode") == "150")
{
reply->deleteLater();

if (m_el < 0 || m_el > 4)
{
emit error(this, "fmt_url_map not found.\nReason: " + QUrl::fromPercentEncoding(url.queryItemValue("reason").replace('+', ' ').toAscii()));

return;
}
else
{
m_el++;
_loadInfo();

return;
}
}
}

emit error(this, "fmt_url_map not found.\nReason: " + QUrl::fromPercentEncoding(url.queryItemValue("reason").replace('+', ' ').toAscii()));
}
else
{
emit error(this, "fmt_url_map not found.\nUnknown reason.");
}
}
}

reply->deleteLater();
}


This is how to use YtInfo:



void MainWindow::_getYtVideo(const QString& id)
{
YtInfo* info = new YtInfo(this);

connect(info, SIGNAL(ready(YtInfo*)), SLOT(ready(YtInfo*)));
connect(info, SIGNAL(error(YtInfo*,QString)), SLOT(error(YtInfo*,QString)));

info->setVideo(id);
info->load();
}


The ready signal is emitted when the info is loaded and the real links is listed.

The other thing that I've used for get the best quality download:



QString YtInfo::getHigherQualityDownload()
{
foreach (YtFormatInfo inf, avaliableFormats())
{
if (m_avaliable.contains(inf.code))
{
return m_avaliable.value(inf.code);
}
}

return "";
}


avaliableFormats() return a list with the YtFormatInfo sorted with having higher quality first. (YtFormatInfo have just a number (to get the real link in the map) and some info that I'm not using now, but will use later).

EDIT:
I've just noticed that downloading the file with C# WebClient works.

My friend tested my real link that I've pasted and it's really working, the problem isn't with the real link or header, so what it can be ?

I've tested and found that my real link works in QMediaPlayer demo. (That demo that uses Phonon)

6425
6426

[ChrisW67]

There are glaringly obvious differences in the two request URIs:


Start of request URI from Opera:
/videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2C itag ...
Start of request URI for your code:
/videoplayback?sparams=id%252Cexpire%252Cip%252Cipb its%252Citag ...

It will almost certainly work better if you do not percent-encode query elements that are already encoded.

[wysota]

It's valid here. I can use it for download here.
Clicking the link in Firefox gives a nice blank page.

[rsilva]

So, we have a problem because I've pasted this link in Opera, Firefox, the Phonon example and Chrome.
Downloaded the file with C# WebClient and sent the link for my friend and ALL places that I've tested worked.

The only problem was inside the Qt App.

But, ChrisW67 was right, that percent encoding problem was the problem and it's working now.
When I paste a link with percent encoding in a browser (or in phonon example) it worked nice.

But now I've found that using percent encoding links to make a request, it encodes the string again.
Maybe that Url does this automatically, or something inside the request.

And in the errorString it doesn't say that this happens, just output the link as the original.

And now, this link without the encoding works.

http://v24.lscache5.c.youtube.com/videoplayback?sparams=id,expire,ip,ipbits,itag,alg orithm,burst,factor&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=B3816D056F902BA5420CECEB5CD560031ED9827C .8153BD9B08D0FC9297019A2FE524297897E4D657&expire=1305658800&key=yt1&ip=189.0.0.0&factor=1.25&id=1adfd74e652d8e6c

(also works in all places that I've tested earlier)

Thanks for the help.

Now, my only problem with this app is the high usage in the item delegate paint with buttons.