Help for downloading Youtube video "server replied: Forbidden"

**rsilva** · 16th May 2011, 23:22

I'm trying to download a video from youtube.

I'm using something like in this python code:
https://github.com/rg3/youtube-dl/bl...ter/youtube-dl

But rewrote it for my code.
Everything works fine, to get the real download link.

I put this link: http://www.youtube.com/watch?v=AGAIq...eature=related

And get this: http://v11.lscache8.c.youtube.com/vi...6008ab9fec6f8a

At here, everything ok, now I'm trying to download the video using this link.
But I got an error in the reply, this error:

"Error downloading http://v11.lscache8.c.youtube.com/vi...6008ab9fec6f8a - server replied: Forbidden"

This is my code:

Qt Code:

Switch view

void Downloader::start()
{
    YtInfo* info = qobject_cast<YtInfo*>((QObject*)m_item->data(Qt::UserRole + 1).toInt());
 
    QNetworkRequest request = QNetworkRequest(QUrl(info->getHigherQualityDownload()));
    //request.setRawHeader("User-Agent", "Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10");
    request.setRawHeader("Youtubedl-no-compression", "True");
 
    //m_reply = m_manager->get(request);
    m_reply = m_manager->head(request);
 
    connect(m_reply, SIGNAL(downloadProgress(qint64,qint64)), SLOT(downloadProgress(qint64,qint64)));
    connect(m_reply, SIGNAL(finished()), SLOT(finished()));
}
 
void Downloader::finished()
{
    qDebug() << m_reply->errorString();
    m_reply->deleteLater();
}

void Downloader::start()
{
    YtInfo* info = qobject_cast<YtInfo*>((QObject*)m_item->data(Qt::UserRole + 1).toInt());

    QNetworkRequest request = QNetworkRequest(QUrl(info->getHigherQualityDownload()));
    //request.setRawHeader("User-Agent", "Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10");
    request.setRawHeader("Youtubedl-no-compression", "True");

    //m_reply = m_manager->get(request);
    m_reply = m_manager->head(request);

    connect(m_reply, SIGNAL(downloadProgress(qint64,qint64)), SLOT(downloadProgress(qint64,qint64)));
    connect(m_reply, SIGNAL(finished()), SLOT(finished()));
}

void Downloader::finished()
{
    qDebug() << m_reply->errorString();
    m_reply->deleteLater();
}

To copy to clipboard, switch view to plain text mode

info->getHigherQualityDownload()
Returns the best quality video avaliable for download.
The link that I've posted.

I've tried changing the User-Agent header, but doesn't work too.

The video link works in the browsers but not inside my Qt Application.

I've tested it in the Qt Browser Demo and the link doesn't work too.
Just got an "non-exiting" page.

But works in Opera, Mozilla and Chrome. (I've not tested it on IE).

The download works even if I'm logged in a youtube account or no.

Someone can help me ?

**wysota** · 17th May 2011, 00:28

So what do you expect us to do? Take a network sniffer and compare the traffic from your app with the one from a full network browser.

**rsilva** · 17th May 2011, 00:36

I'll try it using wireshark.
I've asked because I've thought that someone may have this problem too.
Or someone knows how to solve it, or may be someone knows if this needs header-tricks or something else.
Or a way to find why this is happening, but thanks for answering.

**wysota** · 17th May 2011, 00:54

Look, how would anyone know what is wrong with your code if the whole logic is inside this YtInfo class which we don't have any info about? In general your problem is not Qt related but YouTube related -- you don't know how to download a file from YouTube and not how to do it using Qt. If you were to do it using plain native sockets, you'd have exactly the same problem. By the way the "real" link you pasted is invalid so there is no way of us checking whether anything works.

**rsilva** · 17th May 2011, 03:29

It's valid here. I can use it for download here.
Yeah, I think it's youtube related.

I know how to do this manually, just get this link and use it for download, it's working here.
The only problem is, download this real link with Qt, that doesn't work even with Qt Browser Demo.

In Opera request for the video I have:

Qt Code:

Switch view

Request Method: GET
 
Request URI: /videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=912601&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=40EA572A3CC8F901C65EB029E3F595C090C3F85F.06EDF73184B87C4158D7C0A430408969F43FDC2E&expire=1305612000&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a
 
Request Version: HTTP/1.1
 
User-Agent: Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10\r\n
 
Host: v11.lscache8.c.youtube.com\r\n
 
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1\r\n
 
Accept-Language: pt-BR,pt;q=0.9,en;q=0.8\r\n
 
Accept-Encoding: gzip, deflate\r\n
 
Connection: Keep-Alive\r\n
 
\r\n

Request Method: GET

Request URI: /videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=912601&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=40EA572A3CC8F901C65EB029E3F595C090C3F85F.06EDF73184B87C4158D7C0A430408969F43FDC2E&expire=1305612000&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a

Request Version: HTTP/1.1

User-Agent: Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10\r\n

Host: v11.lscache8.c.youtube.com\r\n

Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1\r\n

Accept-Language: pt-BR,pt;q=0.9,en;q=0.8\r\n

Accept-Encoding: gzip, deflate\r\n

Connection: Keep-Alive\r\n

\r\n

To copy to clipboard, switch view to plain text mode

QNetworkAccessManager request:

Qt Code:

Switch view

Request Method: HEAD
 
Request URI: /videoplayback?sparams=id%252Cexpire%252Cip%252Cipbits%252Citag%252Calgorithm%252Cburst%252Cfactor&fexp=909406%252C912503&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=40EA572A3CC8F901C65EB029E3F595C090C3F85F.06EDF73184B87C4158D7C0A430408969F43FDC2E&expire=1305612000&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a
 
Request Version: HTTP/1.1
 
Youtubedl-no-compression: True\r\n
 
Connection: Keep-Alive\r\n
 
Accept-Encoding: gzip\r\n
 
Accept-Language: pt-BR,en,*\r\n
 
User-Agent: Mozilla/5.0\r\n
 
Host: v11.lscache8.c.youtube.com\r\n
 
\r\n

Request Method: HEAD

Request URI: /videoplayback?sparams=id%252Cexpire%252Cip%252Cipbits%252Citag%252Calgorithm%252Cburst%252Cfactor&fexp=909406%252C912503&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=40EA572A3CC8F901C65EB029E3F595C090C3F85F.06EDF73184B87C4158D7C0A430408969F43FDC2E&expire=1305612000&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a

Request Version: HTTP/1.1

Youtubedl-no-compression: True\r\n

Connection: Keep-Alive\r\n

Accept-Encoding: gzip\r\n

Accept-Language: pt-BR,en,*\r\n

User-Agent: Mozilla/5.0\r\n

Host: v11.lscache8.c.youtube.com\r\n

\r\n

To copy to clipboard, switch view to plain text mode

And now, I've just changed my code to this:

Qt Code:

Switch view

void Downloader::start()
{
    YtInfo* info = qobject_cast<YtInfo*>((QObject*)m_item->data(Qt::UserRole + 1).toInt());
 
    QNetworkRequest request = QNetworkRequest(QUrl(info->getHigherQualityDownload()));
    request.setRawHeader("Accept", "text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1");
    request.setRawHeader("Accept-Encoding", "gzip, deflate");
    request.setRawHeader("Accept-Language", "pt-BR,pt;q=0.9,en;q=0.8");
    request.setRawHeader("User-Agent", "Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10");
 
    m_reply = m_manager->get(request);
 
    connect(m_reply, SIGNAL(downloadProgress(qint64,qint64)), SLOT(downloadProgress(qint64,qint64)));
    connect(m_reply, SIGNAL(finished()), SLOT(finished()));
}

void Downloader::start()
{
    YtInfo* info = qobject_cast<YtInfo*>((QObject*)m_item->data(Qt::UserRole + 1).toInt());

    QNetworkRequest request = QNetworkRequest(QUrl(info->getHigherQualityDownload()));
    request.setRawHeader("Accept", "text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1");
    request.setRawHeader("Accept-Encoding", "gzip, deflate");
    request.setRawHeader("Accept-Language", "pt-BR,pt;q=0.9,en;q=0.8");
    request.setRawHeader("User-Agent", "Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10");

    m_reply = m_manager->get(request);

    connect(m_reply, SIGNAL(downloadProgress(qint64,qint64)), SLOT(downloadProgress(qint64,qint64)));
    connect(m_reply, SIGNAL(finished()), SLOT(finished()));
}

To copy to clipboard, switch view to plain text mode

And I get the same "server replied: Forbidden" again.

This is the request now:

Qt Code:

Switch view

Request Method: GET
 
Request URI: /videoplayback?sparams=id%252Cexpire%252Cip%252Cipbits%252Citag%252Calgorithm%252Cburst%252Cfactor&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=96B89386D0B3947AA32331AD917B7F23AD71AB2C.C8B734A2140BF665BF9A172C45126A38BCD1CCF9&expire=1305622800&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a
 
Request Version: HTTP/1.1
 
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1\r\n
 
Accept-Encoding: gzip, deflate\r\n
 
Accept-Language: pt-BR,pt;q=0.9,en;q=0.8\r\n
 
User-Agent: Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10\r\n
 
Connection: Keep-Alive\r\n
 
Host: v11.lscache8.c.youtube.com\r\n
 
\r\n

Request Method: GET

Request URI: /videoplayback?sparams=id%252Cexpire%252Cip%252Cipbits%252Citag%252Calgorithm%252Cburst%252Cfactor&algorithm=throttle-factor&itag=34&ipbits=8&burst=40&sver=3&signature=96B89386D0B3947AA32331AD917B7F23AD71AB2C.C8B734A2140BF665BF9A172C45126A38BCD1CCF9&expire=1305622800&key=yt1&ip=189.0.0.0&factor=1.25&id=006008ab9fec6f8a

Request Version: HTTP/1.1

Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1\r\n

Accept-Encoding: gzip, deflate\r\n

Accept-Language: pt-BR,pt;q=0.9,en;q=0.8\r\n

User-Agent: Opera/9.80 (Windows NT 5.1; U; pt-BR) Presto/2.8.131 Version/11.10\r\n

Connection: Keep-Alive\r\n

Host: v11.lscache8.c.youtube.com\r\n

\r\n

To copy to clipboard, switch view to plain text mode

With the same request, I'm getting that forbidden error.
What I can do now ? The request is exactly the same.

I'm sending the YtInfo class (ytinfo.h and ytinfo.cpp) in the Attachments.

The main functions is:
That is called with this link:

Qt Code:

Switch view

QString YtInfo::m_videoInfoUrl = "http://www.youtube.com/get_video_info?&video_id=%1%2&ps=default&eurl=&gl=US&hl=en";
 
//%1 = video id
//%2 = "&el" type

QString YtInfo::m_videoInfoUrl = "http://www.youtube.com/get_video_info?&video_id=%1%2&ps=default&eurl=&gl=US&hl=en";

//%1 = video id
//%2 = "&el" type

To copy to clipboard, switch view to plain text mode

Qt Code:

Switch view

void YtInfo::_loadInfo()
{
    if (m_el < 0 || m_el > 4) return;
 
    QList<QString> types;
 
    types.append("");
    types.append("&el=detailpage");
    types.append("&el=embedded");
    types.append("&el=vevo");
 
    QString infoUrl = m_videoInfoUrl.arg(m_id, types.at(m_el));
    QNetworkRequest request = QNetworkRequest(QUrl(infoUrl));
 
    m_manager.get(request);
}
 
void YtInfo::_infoLoaded(QNetworkReply* reply)
{
    if (reply->error())
    {
        emit error(this, reply->errorString());
    }
    else
    {
        QUrl url("http://.../?" + reply->readAll());
 
        if (url.hasQueryItem("fmt_url_map"))
        {
            QString fmtmap = url.queryItemValue("fmt_url_map");
 
            if (!fmtmap.isEmpty())
            {
                fmtmap = QUrl::fromPercentEncoding(fmtmap.toAscii());
                QStringList urlmap = fmtmap.split(',');
 
                foreach (QString fmturl, urlmap)
                {
                    QStringList fmt = fmturl.split('|');
                    m_avaliable[fmt.at(0)] = fmt.at(1);
                }
 
                if (url.hasQueryItem("author"))
                {
                    m_author = url.queryItemValue("author");
                    m_author = QUrl::fromPercentEncoding(m_author.toAscii());
                }
 
                if (url.hasQueryItem("title"))
                {
                    m_title = url.queryItemValue("title");
                    m_title = QUrl::fromPercentEncoding(m_title.replace('+', ' ').toAscii());
                }
 
                if (url.hasQueryItem("length_seconds"))
                {
                    QString tmp = url.queryItemValue("length_seconds");
                    tmp = QUrl::fromPercentEncoding(tmp.toAscii());
 
                    m_length = tmp.toInt();
                }
 
                emit ready(this);
            }
            else
            {
                emit error(this, "Empty fmt_url_map.");
            }
        }
        else
        {
            if (url.hasQueryItem("reason"))
            {
                if (url.hasQueryItem("errorcode"))
                {
                    if (url.queryItemValue("errorcode") == "150")
                    {
                        reply->deleteLater();
 
                        if (m_el < 0 || m_el > 4)
                        {
                            emit error(this, "fmt_url_map not found.\nReason: " + QUrl::fromPercentEncoding(url.queryItemValue("reason").replace('+', ' ').toAscii()));
 
                            return;
                        }
                        else
                        {
                            m_el++;
                            _loadInfo();
 
                            return;
                        }
                    }
                }
 
                emit error(this, "fmt_url_map not found.\nReason: " + QUrl::fromPercentEncoding(url.queryItemValue("reason").replace('+', ' ').toAscii()));
            }
            else
            {
                emit error(this, "fmt_url_map not found.\nUnknown reason.");
            }
        }
    }
 
    reply->deleteLater();
}

void YtInfo::_loadInfo()
{
    if (m_el < 0 || m_el > 4) return;

    QList<QString> types;

    types.append("");
    types.append("&el=detailpage");
    types.append("&el=embedded");
    types.append("&el=vevo");

    QString infoUrl = m_videoInfoUrl.arg(m_id, types.at(m_el));
    QNetworkRequest request = QNetworkRequest(QUrl(infoUrl));

    m_manager.get(request);
}

void YtInfo::_infoLoaded(QNetworkReply* reply)
{
    if (reply->error())
    {
        emit error(this, reply->errorString());
    }
    else
    {
        QUrl url("http://.../?" + reply->readAll());

        if (url.hasQueryItem("fmt_url_map"))
        {
            QString fmtmap = url.queryItemValue("fmt_url_map");

            if (!fmtmap.isEmpty())
            {
                fmtmap = QUrl::fromPercentEncoding(fmtmap.toAscii());
                QStringList urlmap = fmtmap.split(',');

                foreach (QString fmturl, urlmap)
                {
                    QStringList fmt = fmturl.split('|');
                    m_avaliable[fmt.at(0)] = fmt.at(1);
                }

                if (url.hasQueryItem("author"))
                {
                    m_author = url.queryItemValue("author");
                    m_author = QUrl::fromPercentEncoding(m_author.toAscii());
                }

                if (url.hasQueryItem("title"))
                {
                    m_title = url.queryItemValue("title");
                    m_title = QUrl::fromPercentEncoding(m_title.replace('+', ' ').toAscii());
                }

                if (url.hasQueryItem("length_seconds"))
                {
                    QString tmp = url.queryItemValue("length_seconds");
                    tmp = QUrl::fromPercentEncoding(tmp.toAscii());

                    m_length = tmp.toInt();
                }

                emit ready(this);
            }
            else
            {
                emit error(this, "Empty fmt_url_map.");
            }
        }
        else
        {
            if (url.hasQueryItem("reason"))
            {
                if (url.hasQueryItem("errorcode"))
                {
                    if (url.queryItemValue("errorcode") == "150")
                    {
                        reply->deleteLater();

                        if (m_el < 0 || m_el > 4)
                        {
                            emit error(this, "fmt_url_map not found.\nReason: " + QUrl::fromPercentEncoding(url.queryItemValue("reason").replace('+', ' ').toAscii()));

                            return;
                        }
                        else
                        {
                            m_el++;
                            _loadInfo();

                            return;
                        }
                    }
                }

                emit error(this, "fmt_url_map not found.\nReason: " + QUrl::fromPercentEncoding(url.queryItemValue("reason").replace('+', ' ').toAscii()));
            }
            else
            {
                emit error(this, "fmt_url_map not found.\nUnknown reason.");
            }
        }
    }

    reply->deleteLater();
}

To copy to clipboard, switch view to plain text mode

This is how to use YtInfo:

Qt Code:

Switch view

void MainWindow::_getYtVideo(const QString& id)
{
    YtInfo* info = new YtInfo(this);
 
    connect(info, SIGNAL(ready(YtInfo*)), SLOT(ready(YtInfo*)));
    connect(info, SIGNAL(error(YtInfo*,QString)), SLOT(error(YtInfo*,QString)));
 
    info->setVideo(id);
    info->load();
}

void MainWindow::_getYtVideo(const QString& id)
{
    YtInfo* info = new YtInfo(this);

    connect(info, SIGNAL(ready(YtInfo*)), SLOT(ready(YtInfo*)));
    connect(info, SIGNAL(error(YtInfo*,QString)), SLOT(error(YtInfo*,QString)));

    info->setVideo(id);
    info->load();
}

To copy to clipboard, switch view to plain text mode

The ready signal is emitted when the info is loaded and the real links is listed.

The other thing that I've used for get the best quality download:

Qt Code:

Switch view

QString YtInfo::getHigherQualityDownload()
{
    foreach (YtFormatInfo inf, avaliableFormats())
    {
        if (m_avaliable.contains(inf.code))
        {
            return m_avaliable.value(inf.code);
        }
    }
 
    return "";
}

QString YtInfo::getHigherQualityDownload()
{
    foreach (YtFormatInfo inf, avaliableFormats())
    {
        if (m_avaliable.contains(inf.code))
        {
            return m_avaliable.value(inf.code);
        }
    }

    return "";
}

To copy to clipboard, switch view to plain text mode

avaliableFormats() return a list with the YtFormatInfo sorted with having higher quality first. (YtFormatInfo have just a number (to get the real link in the map) and some info that I'm not using now, but will use later).

EDIT:
I've just noticed that downloading the file with C# WebClient works.

My friend tested my real link that I've pasted and it's really working, the problem isn't with the real link or header, so what it can be ?

I've tested and found that my real link works in QMediaPlayer demo. (That demo that uses Phonon)

ytinfo.cpp
ytinfo.h

**ChrisW67** · 17th May 2011, 06:34

There are glaringly obvious differences in the two request URIs:

Start of request URI from Opera:
/videoplayback?sparams=id%2Cexpire%2Cip%2Cipbits%2Citag ...
Start of request URI for your code:
/videoplayback?sparams=id%252Cexpire%252Cip%252Cipbits%252Citag ...

It will almost certainly work better if you do not percent-encode query elements that are already encoded.

**wysota** · 17th May 2011, 10:41

Originally Posted by rsilva

It's valid here. I can use it for download here.

Clicking the link in Firefox gives a nice blank page.

**rsilva** · 17th May 2011, 13:57

So, we have a problem because I've pasted this link in Opera, Firefox, the Phonon example and Chrome.
Downloaded the file with C# WebClient and sent the link for my friend and ALL places that I've tested worked.

The only problem was inside the Qt App.

But, ChrisW67 was right, that percent encoding problem was the problem and it's working now.
When I paste a link with percent encoding in a browser (or in phonon example) it worked nice.

But now I've found that using percent encoding links to make a request, it encodes the string again.
Maybe that Url does this automatically, or something inside the request.

And in the errorString it doesn't say that this happens, just output the link as the original.

And now, this link without the encoding works.

http://v24.lscache5.c.youtube.com/vi...dfd74e652d8e6c

(also works in all places that I've tested earlier)

Thanks for the help.

Now, my only problem with this app is the high usage in the item delegate paint with buttons.

Thread: Help for downloading Youtube video "server replied: Forbidden"

Thread Tools

Search Thread

Display

Help for downloading Youtube video "server replied: Forbidden"

Re: Help for downloading Youtube video "server replied: Forbidden"

Re: Help for downloading Youtube video "server replied: Forbidden"

Re: Help for downloading Youtube video "server replied: Forbidden"

Re: Help for downloading Youtube video "server replied: Forbidden"

Re: Help for downloading Youtube video "server replied: Forbidden"

Re: Help for downloading Youtube video "server replied: Forbidden"

Re: Help for downloading Youtube video "server replied: Forbidden"

Similar Threads

Downloading a video from youtube

.pro project file includes "unix" with "macx" defintions with qmake

Translation QFileDialog standart buttons ("Open"/"Save"/"Cancel")

question about "fortune server"

How a server can write "Hello" to a browser ?

Bookmarks

Bookmarks

Posting Permissions