Comparing 33C3 and 34C3 tweets

Distribution of tweet lengths

Usually each tweet about any subject consists at least of 2000 characters. This might surprise you, because a tweet-text used to be 140 characters in length, until Twitter increased it to 280 characters in 2017. But a tweet contains a lot of metadata, and this makes a tweet exceed 2000 characters in length when serialized to a compact JSON string (not pretty-printed).

Line lengths of serialized tweet data-structures can be determined with:

awk '{print length}' stream_of_tweets.json > 34c3_strlength.txt .

The following histograms show that there were many more tweet data-structures for the 34C3 congress than for the 33C3 congress. There are also many junk tweet-fragments, from 0-2500 characters in length, that are not present in the 33C3 histogram (red rectangle).

I think the multi-modal peak structure of the histograms represents tweets that have been retweeted once or twice, or even more often. Actually I haven’t investigated that.

For 34C3, the harvesting script was not able to extract all tweets as syntactically valid JSON files. Visual inspection revealed that for some unknown reason they consist of many corrupted and duplicated tweets. I have not figured out exactly what was wrong. It seems that many tweets were nested inside each other randomly, but the nesting occured almost always on a single line. Still, some tweet fragments from the 34C3 scrape did span multiple lines; see the red rectangle in the figure above. However, there were so many garbage tweets in the 600MB .json-file that it was impossible to remove them all manually.

Bad JSON

So, what did the JSON look like? Not like this:

tweet1
tweet2
tweet3
tweet4

…but like this:

tweet1_tweet2fragment1
tweet2fragment2
tweet3
tweet4

Using a series of shell commands and pipelines, I tried my best to recover the first tweet from each line, discarding the fragmented JSON objects tweets misplaced and appended to each line. This recovery step might produce duplicated tweets for some reason.

Intermediate result:

tweet1
tweet1
fragment2
tweet3
tweet4

Valid tweets

Valid tweets always start with ‘“{created_at:”’, at the beginning of the line.

• keep these
• remove duplicated tweets

Thus I had to further preprocess the tweets with a command similar to this pipe:

grep  "^{\"created_at" stream__34c3.id_date_text.json | 
   uniq > 34c3.created_at.json`

Convert dates to strings, group them into day-of-month, and hours-of-day.

Tweets gathered during the two events:

There exist still more tweets about the 34C3 conference than about the 33C3, but the difference in tweet count is no longer as big as the first histogram above suggested.

Plot: User activity over time

This plot compares user activity on twitter during the 2 conferences, one year apart:

For the 33C3 I’ve started gathering tweets later (on Dec 27, 2016) and finished earlier, after 6 days. For the 34C3 I’ve let the script run unattended for 13 days (see also ‘Extra plots’, at end of post).

Observations (TBC)

The 34C3 congress had 15000 visitors, much more than 33C3 (12000), but absolute count of tweets per hour seems to be lower in 2017, during the first 2 days, whereas during the last 2 days users seem to have tweeted a bit more, though.

• Maybe Twitter has become less popular among the visitors?
• Maybe my preprocessing script was too aggressive in cleaning up things.
• Maybe the Twitter streaming API works differently?

(To be continued:)

• Wordcloud?
• Network Diagrams from mentions and retweets?