Published on

No-Code TikTok API Data Scraping with HAR Files (Still Works in 2024)

Introduction

TikTok data, particularly information about influencers, hashtags, and posts, is immensely valuable for digital marketers looking to match influencers with brands seeking promotions. If you're intent on collecting public data from TikTok but feel thwarted by technical roadblocks like signed requests, don’t worry. There are ways to navigate through these challenges.

TikTok makes it intentionally difficult to scrape its data using automated tools, but we're here to highlight some effective methodologies that still respect their terms of service.

TikTok's Signed Requests and Their Challenges

TikTok employs signed requests to protect its data. These are essentially digital signatures that validate requests to their servers. If anything changes in the request URL, such as the hashtag ID, the request will fail unless a new signature is generated.

First, inspect the network traffic on TikTok's website to locate the data you need. By switching to the network tab and refreshing the page, you can identify where posts are loaded from and see their associated data.

Using Network Inspection to Identify Data

  1. Right-click on the TikTok page and select Inspect.
  2. Click on the Network tab and refresh the page.
  3. Sort network traffic by size to identify the largest payloads likely containing the data.
  4. Inspect these payloads to find structured data on posts, author information, and hashtags.

Bypassing Signed Requests

Using the URL identified directly on TikTok's network tab doesn't work due to signed requests. These requests validate with TikTok servers to ensure authenticity, preventing arbitrary data access through tools like curl.

Selenium to the Rescue

An unofficial approach uses Selenium, a tool to simulate a real web browser, to generate the necessary signatures. Some solutions use Selenium to bypass the signed requests, though this can be complex and potentially against TikTok's terms of service.

My Solution: Using HAR Files for Manual Data Collection

A safer and easier method to collect TikTok data is using HAR (HTTP Archive) files.

  1. Manual Browsing: Manually browse TikTok, adhering fully to their terms of service.
  2. HAR File Export: After gathering the required data, go to the Network tab and click Export HAR to save all network traffic.
  3. Data Parsing Tool: Use a freemium tool to parse the HAR file. This groups together similar requests and lets you access the raw JSON data.

For those interested, a paid plan simplifies the process:

  • Automatically combines JSON data from all observed posts.
  • Provides options to download the data as a CSV file for easier handling.

This method is compliant with TikTok's terms as it does not use automated scraping tools, minimizing the risk of being banned or facing legal repercussions.

Conclusion

TikTok’s signed requests aim to prevent automated data scraping, but you can still collect valuable public data manually. Using HAR files is a compliant and effective way to gather data without resorting to sketchy tactics that violate terms of service.

Keywords

  • TikTok
  • Data Scraping
  • Signed Requests
  • Network Inspection
  • Selenium
  • HAR Files
  • JSON
  • CSV
  • Manual Data Collection

FAQ

Q: What is a signed request on TikTok? A: Signed requests are digital signatures used by TikTok to validate requests to their servers and ensure their authenticity.

Q: How can I inspect network traffic on TikTok? A: Right-click on the page, select Inspect, open the Network tab, and refresh the page to identify where data on posts is loaded from.

Q: Can I use an unofficial API to scrape TikTok data? A: While unofficial APIs using tools like Selenium exist, they may violate TikTok's terms of service and are not guaranteed to work long-term.

Q: What are HAR files and how can they help in data collection? A: HAR files record web traffic data during your browsing session. Exporting and parsing these files provides access to the raw JSON data from TikTok, which you can compile into a CSV for easier analysis.

Q: Is manual data collection compliant with TikTok's terms of service? A: Yes, manually browsing TikTok and exporting a HAR file while using the platform according to its terms of service is a compliant way to collect data.

Read More