Community Highlights Archive Data

git submodule used by the Community Highlights Archive site builder

Note: this repo is not for making suggestions for content to be highlighted, it is for content that has already been highlighted.

Contributing

Meta Values

The meta value are associated with images manually, automated no image processing applied.

Sometimes meta values are added in response to something featured in a new Community Highlights video
Sometimes aspects of a Community Highlights entry weren't noticed 🤷‍♂️

Notes

Please edit data/highlights-meta-to-add.json rather than making changes to the various data files as portions of that process are already handled by the site builder.
If you are not familiar with editing files on GitHub, please refer to GitHub's documentation before attempting to edit data/highlights-meta-to-add.json.
Meta values are generally lower-case, community names preferred over actual names (e.g. "bean" instead of "Space Giraffe-Tick-Penguin-Whale Thing" or "Confusing Creature")

Renaming an existing value

Please don't 😅. Please instead file an Issue to have a meta value changed.

Adding meta values to images

Locate or add the JSON key for the new or current meta value, using a JSON array ([] for new values)
Find the image URL
- The simplest way to check where an image came from may be to browse the Community Highlights Archive website. If the highlights entry does not appear to have a source link (i.e. if it was extracted from the YouTube video), you may need to contact SignpostMarv on Discord or Twitter for the yt-dlp-style image link.
- For Tweets:
  1. open the tweet (e.g. https://twitter.com/jembawls/status/1383101151814488068)
  2. open the image in the tweet that was used in the highlights entry in a new tab (e.g. https://pbs.twimg.com/media/EzHDHAOXEAo-Y3T?format=jpg&name=small)
  3. Search the GitHub repo for the image id - the bit after /media/ but before ?format (e.g. https://github.com/search?q=repo%3ASatisfactory-Clips-Archive%2FCommunity-Highlights-Archive-Data%20EzHDHAOXEAo-Y3T&type=code)
  4. Copy the image url from the search results (e.g. https://pbs.twimg.com/media/EzHDHAOXEAo-Y3T.jpg)
- For Reddit posts (similar to the above Tweets example):
  1. Open the Reddit post
  2. Open the image from the Reddit post that was featured in the highlights entry in a new tab
  3. Search for the filename
  4. Copy the image url from the search results
Add the image URL to the appropriate JSON array
Repeat steps 1-3 for each change to be made
Save/Commit/PR

Example

{
	"some new meta value": [
		"https://i.reddit.it/some-reddit-image.png",
		"https://pbs.twimg.com/media/some-twitter-image.jpg",
		"yt-dlp/some-youtube-id@timestamp.png"
	],
	"some other meta value": [
		"https://i.reddit.it/some-reddit-image.png",
		"https://pbs.twimg.com/media/some-twitter-image.jpg",
		"yt-dlp/some-youtube-id@timestamp.png"
	]
}

files

`data/by-youtube-video`

YouTube video descriptions with chapter markers are in sub-folders for their YouTube video id with the prefix yt-, with the filename desc.txt e.g. data/by-youtube-video/yt-Ny6KcOCdABM/desc.txt

For Community Highlights not sourced from the Community Highlights Archive YouTube channel, the contents of desc.txt is preferred to match the format of YouTube chapter markers rather than the original video's description in the event there are no such chapter markers.

`data/highlights-meta/`

`data/highlights-meta/highlights-meta-history/`

Files in here represent the state of data/highlights-meta.json at a given point in its history. These are used by the gui in the site builder to flag which meta values were added since a given image was last signed off.

`data/highlights-meta/twitter-tweet-manual-entry/`

Since the Twitter api changes, retrieval of Twitter api data is now a manual process.

The files are processed by the site builder to fill in the gaps of the lost functionality.
Filenames are the lowercase hexadecimal digest of the sha-512 hash of the tweet url.

Example

Has images

https://twitter.com/SoenKigi/status/1599976711445131264 resolves to 8148ef2b37d3cf8b72f10e05047e4b6833f8ce3e9b29f5fdffbb4da0b64269eadf34c68e20f23eb8e14556e9534b0cff4dc02eea6aa4902488493eb37ad02c5d.json

{
	"includes": {
		"media": [
			{
				"type": "photo",
				"url": "https://pbs.twimg.com/media/FjRCVs5XgAMOWH6.jpg"
			}
		]
	}
}

Tweet deleted since sourcing

{
	"includes": {
		"media": []
	}
}

`data/highlights-meta/grouped-meta.json`

Groups the meta into a more user-friendly hierarchical order. Also used by the site builder to change the sort order of data/highlights-meta.json.

`data/highlights-meta/grouped-meta.schema.json`

Used for validating changes against grouped-meta.json, can be used by IDEs for auto-completion when editing data/highlights-meta/grouped-meta.json.

`data/highlights-meta/image-source-signoff-done.json`

Used by the gui in the site builder to track which entry in data/highlights-meta/highlights-meta-history/ an image was signed-off against.

`data/highlights-meta/image-source-signoff-pending.json`

Used by the gui in the site builder to flag which images are pending sign off against the latest version of data/hlights-meta.json.

`data/highlights-meta/schema-ersion-history.json`

Indicates when each version of data/highlights-meta.json was created. Generally corresponds to a file in data/highlights-meta/highlights-meta-history/, but some files may be absent.

`data/authors.json`

An index of Community Highlights creators.

Validated by data/authors.schema.json.

`data/authors.json#unaliased`

The names extracted from data/by-youtube-video/*/desc.txt with the uuid generated randomly at the time they are processed by the site builder.

`data/authors.json#aliased`

Creator names are not always consistently presented in data/by-youtube-video/*/desc.txt, due to (but not limited to):

transcription error (one cannot generally copy/paste text from a video 🤷‍♂️)
changes in casing or spelling across various social networks
changes in identity or rebranding

Entries can be added to this section in advance of their corresponding file being added to data/by-youtube-video/

`data/authors.schema.json`

The JSON Schema for validating data/authors.json.

`data/dated.json`

An index of Community Highlights videos, listed by date.

Validated by data/dated.schema.json.

`data/dated.schema.json`

The JSON Schema for validating data/dated.json.

`data/discord-images.json`

An index of Community Highlights links, the corresponding image & any appropriate caption or meta values.

Validated by data/image-sources.schema.json.

`data/has-author-but-no-links.json`

Originally intended to track Community Highlights that have a listed creator but a link for the entry was not found.

Can also be used for:

Community Highlights entries that had links at the time of being "added to the vault" but the link had been deleted by the time the video had been processed into the site builder.
Community Highlights entries that have additional photoshopping added for humour by the host(s) of the Community Highlights segment of the dev stream.
Community Highlights entries that have particular attention drawn to them (usually by zooming in etc.).

Entries are listed by date, then video id, then by YouTube chapter marker.

Validated by data/yt-dlp.schema.json.

`data/has-author-but-no-links.json#//*/timestamp`

The point in the video at which this entry starts.

`data/has-author-but-no-links.json#//*/raw_text`

The line in the description without the chapter marker prefix.

`data/has-author-but-no-links.json#//*/timestamps_for_links`

Points in the source video to extract images from. To be deprecated eventually.

`data/has-author-but-no-links.json#//*/highlightsMeta`

A list of objects representing the images for this entry & their corresponding meta values.

`data/highlights-meta.json`

The JSON Schema file for the highlights meta values.

Processed by the site builder to keep the values in the order defined in data/highlights-meta/grouped-meta.json.

`data/highlights-meta-to-add.json`

A key-value pair of new meta values and the corresponding images to apply them to.

Utilised by the site builder to defer amendments to data/highlights-meta.json and data/highlights-meta/grouped-meta.json as this action will lead to all file references in data/highlights-meta/image-source-signoff-done.json being flagged as out-of-date and resetting the contents of data/highlights-meta/image-source-signoff-pending.json.

`data/image-sources.schema.json`

The JSON Schema for validating:

data/discord-images.json
data/reddit-images.json
data/twitter-images.json
data/youtube-images.json

`data/reddit-images.json`

An index of Community Highlights links, the corresponding image & any appropriate caption or meta values.

Validated by data/image-sources.schema.json.

`data/twitch-clip-image.json`

An index of Community Highlights links, the corresponding image & any appropriate caption or meta values.

Validated by data/twitch-clip-images.schema.json.

`data/twitch-clip-images.json`

The JSON Schema for validating data/twitch-clip-images.json.

`data/twitter-images.json`

An index of Community Highlights links, the corresponding image & any appropriate caption or meta values.

Validated by data/image-sources.schema.json.

`data/youtube-images.json`

An index of Community Highlights links, the corresponding image & any appropriate caption or meta values.

Validated by data/image-sources.schema.json.

`data/yt-dlp.schema.json`

The JSON Schema for validating data/has-author-but-no-links.json.

Files

README.md

Latest commit

History

README.md

File metadata and controls

Community Highlights Archive Data

Contributing

Meta Values

Notes

Renaming an existing value

Adding meta values to images

Example

files

data/by-youtube-video

data/highlights-meta/

data/highlights-meta/highlights-meta-history/

data/highlights-meta/twitter-tweet-manual-entry/

Example

Has images

Tweet deleted since sourcing

data/highlights-meta/grouped-meta.json

data/highlights-meta/grouped-meta.schema.json

data/highlights-meta/image-source-signoff-done.json

data/highlights-meta/image-source-signoff-pending.json

data/highlights-meta/schema-ersion-history.json

data/authors.json

data/authors.json#unaliased

data/authors.json#aliased

data/authors.schema.json

data/dated.json

data/dated.schema.json

data/discord-images.json

data/has-author-but-no-links.json

data/has-author-but-no-links.json#*/*/*/timestamp

data/has-author-but-no-links.json#*/*/*/raw_text

data/has-author-but-no-links.json#*/*/*/timestamps_for_links

data/has-author-but-no-links.json#*/*/*/highlightsMeta

data/highlights-meta.json

data/highlights-meta-to-add.json

data/image-sources.schema.json

data/reddit-images.json

data/twitch-clip-image.json

data/twitch-clip-images.json

data/twitter-images.json

data/youtube-images.json

data/yt-dlp.schema.json

`data/by-youtube-video`

`data/highlights-meta/`

`data/highlights-meta/highlights-meta-history/`

`data/highlights-meta/twitter-tweet-manual-entry/`

`data/highlights-meta/grouped-meta.json`

`data/highlights-meta/grouped-meta.schema.json`

`data/highlights-meta/image-source-signoff-done.json`

`data/highlights-meta/image-source-signoff-pending.json`

`data/highlights-meta/schema-ersion-history.json`

`data/authors.json`

`data/authors.json#unaliased`

`data/authors.json#aliased`

`data/authors.schema.json`

`data/dated.json`

`data/dated.schema.json`

`data/discord-images.json`

`data/has-author-but-no-links.json`

`data/has-author-but-no-links.json#//*/timestamp`

`data/has-author-but-no-links.json#//*/raw_text`

`data/has-author-but-no-links.json#//*/timestamps_for_links`

`data/has-author-but-no-links.json#//*/highlightsMeta`

`data/highlights-meta.json`

`data/highlights-meta-to-add.json`

`data/image-sources.schema.json`

`data/reddit-images.json`

`data/twitch-clip-image.json`

`data/twitch-clip-images.json`

`data/twitter-images.json`

`data/youtube-images.json`

`data/yt-dlp.schema.json`