Unavailable images/videos from API which are actually present in twitter status

[simsausaurabh]

• Issue type: Bug report

Short description

Currently the search api with filters do not provide correct data.

e.g.

Link: https://api.loklak.org/api/search.json?q=from:MastercardDE&minified=true&source=all&maximumRecords=20&timezoneOffset=-330&startRecord=1&filter=video

returns data:

{
  "readme_0": "THIS JSON IS THE RESULT OF YOUR SEARCH QUERY - THERE IS NO WEB PAGE WHICH SHOWS THE RESULT!",
  "readme_1": "loklak.org is the framework for a message search system, not the portal, read: http://api.loklak.org/about.html#notasearchportal",
  "readme_2": "This is supposed to be the back-end of a search portal. For the api, see http://api.loklak.org/api.html",
  "readme_3": "Parameters q=(query), source=(cache|backend|twitter|all), callback=p for jsonp, maximumRecords=(message count), minified=(true|false)",
  "search_metadata": {
    "startRecord": 1,
    "maximumRecords": 20,
    "count": 2,
    "hits": 2,
    "period": 689991001,
    "query": "from:MastercardDE",
    "filter": "video",
    "client": "47.8.168.216",
    "time": 984,
    "servicereduction": "false",
    "count_twitter_all": 0,
    "count_twitter_new": 2,
    "count_backend": 0,
    "cache_hits": 0,
    "scraperInfo": "local",
    "index": "messages_day"
  },
  "statuses": [
    {
      "provider_type": "SCRAPED",
      "audio_count": 0,
      "hashtags": [
        "gpay",
        "Mastercard"
      ],
      "hashtags_count": 2,
      "favourites_count": 13,
      "link": "https://twitter.com/MastercardDE/status/1014851961852751872",
      "created_at": "2018-07-05T12:42:17.000Z",
      "videos": [],
      "mentions_count": 2,
      "without_lu_len": 108,
      "text_length": 108,
      "retweet_count": 8,
      "unshorten": {},
      "without_l_len": 108,
      "screen_name": "MastercardDE",
      "id_str": "1014851961852751872",
      "links_count": 0,
      "links": [],
      "videos_count": 0,
      "text": "Smartphone mit Google Pay gestohlen? Mobiles Bezahlen ist sicher! 💳📲 #gpay #Mastercard @GoogleDE @GoogleDE",
      "audio": [],
      "place_id": "",
      "timestamp": "2018-07-07T12:55:29.440Z",
      "classifier_language_probability": 1.709835E-10,
      "timestamp_id": 1530968128933,
      "place_name": "",
      "images": [],
      "without_luh_len": 108,
      "classifier_language": "german",
      "hosts": [],
      "images_count": 0,
      "source_type": "TWITTER",
      "hosts_count": 0,
      "place_context": "ABOUT",
      "mentions": [
        "GoogleDE",
        "GoogleDE"
      ],
      "user": {
        "appearance_first": "2018-07-07T12:55:28.933Z",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/753607884924157952/ZGW7ReKQ_bigger.jpg",
        "screen_name": "MastercardDE",
        "user_id": "859373310",
        "name": "MastercardDE",
        "appearance_latest": "2018-07-07T12:55:28.933Z"
      }
    },
    {
      "provider_type": "SCRAPED",
      "audio_count": 0,
      "hashtags": [
        "Mastercard",
        "GooglePay"
      ],
      "hashtags_count": 2,
      "favourites_count": 14,
      "link": "https://twitter.com/MastercardDE/status/1011957930902671360",
      "created_at": "2018-06-27T13:02:26.000Z",
      "videos": [],
      "mentions_count": 2,
      "without_lu_len": 118,
      "text_length": 118,
      "retweet_count": 6,
      "unshorten": {},
      "without_l_len": 118,
      "screen_name": "MastercardDE",
      "id_str": "1011957930902671360",
      "links_count": 0,
      "links": [],
      "videos_count": 0,
      "text": "Bezahlen mit dem Android-Smartphone und #Mastercard - sicher, schnell und einfach. #GooglePay @GoogleDE 💳📲 @GoogleDE",
      "audio": [],
      "place_id": "",
      "timestamp": "2018-07-07T12:55:29.440Z",
      "classifier_language_probability": 2.379756E-11,
      "timestamp_id": 1530968128932,
      "place_name": "",
      "images": [],
      "without_luh_len": 118,
      "classifier_language": "german",
      "hosts": [],
      "images_count": 0,
      "source_type": "TWITTER",
      "hosts_count": 0,
      "place_context": "ABOUT",
      "mentions": [
        "GoogleDE",
        "GoogleDE"
      ],
      "user": {
        "appearance_first": "2018-07-07T12:55:28.933Z",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/753607884924157952/ZGW7ReKQ_bigger.jpg",
        "screen_name": "MastercardDE",
        "user_id": "859373310",
        "name": "MastercardDE",
        "appearance_latest": "2018-07-07T12:55:28.933Z"
      }
    }
  ],
  "aggregations": {}
}

and in first status it does not shows any video count and video link. But in actual status on twitter: https://twitter.com/MastercardDE/status/1011957930902671360, there is a video (i.e. status contains a video).

Environment

• Operating system: Ubuntu 16.04 LTS
• Software version:
• Software source:

Steps to reproduce

1. Search something on api.loklak and apply filters to compare with the actual status.

Expected behaviour

It should contain the video/image that is present in actual status on twitter.

Actual behaviour

Currently it only shows the statuses which contain the image/video but do not provide the actual vidoes/images in all cases.