App scraping - get data from android apps

Hello everyone,

I have a few questions regarding on how apps communicate and exchange data. The knowledge from the web scraping course did not help because it seems to be much more complex. I have a few ideas for future projects on data which are only accessible by App and not Browser. I’ve seen that apps can use HTML requests too, but sometimes the communication path is more intertwined.

Here is what I’ve done or what I need to know:

1. Setting up the required tools for analyzing app communication on Windows 10

I use an android emulator (Nox Player or BlueStack) to use the app on Windows. Afterwards I analyze the network traffic with Wireshark (Network Monitor) to identify exchanged packages.
Questions left:

  • a) It seems to be possible for some Apps to use simple requests (like in the web scraping course) but other have a more hidden communication. I don’t know much about network communication. So I have problems to understand what is going on if I only get communication with protocols like TCP, DNS, TLS. If I do a follow-up analysis of some of the DNS or TCP path I get something that looks like a list of numbered packages where only the headers are readable but the content is encrypted. It is possible to get this header word snippets if I decode it in YAML.
    Are there any possibilities to better understand what is going on there and to get usable data?

  • b) Even if I understand how the app is communicating, is it possible to use it for automated requests to track data like normal web scraping? How should I do that the best way?

  • c) Will there be any problems with authentication or authorization etc.? Because there is no official API or scrapable displayed information.

2. Are there any problems with the law in relation of taking data from the app?

Maybe it is like web scraping? The data are visible for everyone.

  • Are there any problems with taking data I can see in the app anyway?

3. An example what it could be used for:

Maybe for tracking cars of a car sharing app. To make analysis of the movement profiles in a city (hot spots, time-related aspects,…) or something like that. The information of where a car is, is mostly for every app user visible if you make your location available. So it is a public information?

I hope I have done everything right in my first post and someone can help me. I work for days on this problem and have no solution.

Best wishes
Jan