[BUG] AnkiDroid API no longer lists media on cards

[piotrbrzezina]

Hi, I'm trying to write an app that will launch upon unlocking the phone, and its only task will be to present one card, then close after receiving a response. Unfortunately, I'm struggling with one problem, namely how to display images and play audio files because I don't have access to them due to the new Android permissions. I'm wondering if there's a way around this?

I was thinking about whether it would be possible to add another method to the AnkiDroid API that, return an image/audio binary data if it had access to such a file. Do you think this is doable in Android?

[welcome]

Hello! 👋 Thanks for logging this issue. Please remember we are all volunteers here, so some patience may be required before we can get to the issue. Also remember that the fastest way to get resolution on an issue is to propose a change directly, https://github.com/ankidroid/Anki-Android/wiki/Contributing

[mikehardy]

Hey there - that seems feasible. We already have a ContentResolver - it allows you to fetch a card IIRC, it could also allow you to fetch an array (or whatever ContentResolver calls it - IIRC it is kind of a database-ish design so maybe a "result set" or whatever) of all media related to that card

I imagine you'd want file name, media object size then the data stream or similar

Anything that did something like that seems reasonable to me

[piotrbrzezina]

Thanks @mikehardy for the reply, but unfortunately, I'm not sure if I understood it correctly (I'm not an Android developer, I've only learned a bit about it). Are you saying that I can already retrieve image data using the AnkiDroid API, or did you mean that a new method needs to be added to the API, which would return file data for a specific card?

[mikehardy]

@piotrbrzezina I should have looked more closely before, and I'm sorry I was ambiguous.

It appears that media files are on the ReviewInfo object, which sounds like it might be just what you need:

Anki-Android/api/src/main/java/com/ichi2/anki/FlashCardsContract.kt

Line 688 in 83ed107

* JSONArray | MEDIA_FILES | read-only | The media files, like images and sound files, contained in the cards.

[piotrbrzezina]

Thanks for the information. I was trying to do what you proposed, but in my case, it is not working. In that JSON, I only get an empty array '[]'. Could you check if I'm doing it correctly? This is the code I used to get data from the JSON array.

          val deckSelector = "limit=?,deckID=?"
          val deckArguments = arrayOfNulls<String>(2)
          deckArguments[0] = "" + 1
          deckArguments[1] = "" + deckId
          
          val card = HashMap<String, String>()
          val cardCursor = context.contentResolver.query(
                  FlashCardsContract.ReviewInfo.CONTENT_URI,
                  null,
                  deckSelector,
                  deckArguments,
                  null
          )
          
          if (cardCursor?.moveToFirst() == true) {
              val noteIdIndex = cardCursor.getColumnIndex(FlashCardsContract.ReviewInfo.NOTE_ID)
              val cardIdIndex = cardCursor.getColumnIndex(FlashCardsContract.ReviewInfo.CARD_ORD)
              val buttonCountIndex = cardCursor.getColumnIndex(FlashCardsContract.ReviewInfo.BUTTON_COUNT)
              val nextReviewIndex = cardCursor.getColumnIndex(FlashCardsContract.ReviewInfo.NEXT_REVIEW_TIMES)
              val mediaFilesIndex = cardCursor.getColumnIndex(FlashCardsContract.ReviewInfo.MEDIA_FILES)
          
              if (noteIdIndex > -1 && cardIdIndex > -1) {
                  val noteId = cardCursor.getString(noteIdIndex)
                  val specificCardId = cardCursor.getString(cardIdIndex)
                  val buttonCount = cardCursor.getString(buttonCountIndex)
                  val reviewIndex = cardCursor.getString(nextReviewIndex)
                  val mediaFiles = cardCursor.getString(mediaFilesIndex)

Maybe the issue is that the deck I'm trying to work with is not following some guidelines?

[mikehardy]

Sorry - I honestly don't know, this may be a case of needing to get AnkiDroid building locally and adding debug code into the part of AnkiDroid that populates (or should populate) the media files, and perhaps doing the same with the api that gets built into your app. I apologize but I won't have time to pursue this myself, so getting a locally test/debug environment for the other side of the API seems like fastest path forward

[david-allison]

Relevant:

Anki-Android/AnkiDroid/src/main/java/com/ichi2/anki/provider/CardContentProvider.kt

Line 1077 in 2910ce5

FlashCardsContract.ReviewInfo.MEDIA_FILES -> rb.add(JSONArray(col.media.filesInStr(currentCard.question(col) + currentCard.answer(col))))

Also not going to have time to dig deep here, apologies!

Anki-Android/AnkiDroid/src/main/java/com/ichi2/libanki/Media.kt

Lines 60 to 72 in c63e93a

	/**
	* Extract media filenames from an HTML string.
	*
	* @param string The string to scan for media filenames ([sound:...] or <img...>).
	* @return A list containing all the sound and image filenames found in the input string.
	*/
	fun filesInStr(string: String): List<String> {
	return col.backend.extractAvTags(string, true).avTagsList.filter {
	it.hasSoundOrVideo()
	}.map {
	it.soundOrVideo
	}
	}

[david-allison]

My presumption is that filesInStr was broken in 7a65160

I suspect it previously returned images and audio/video, and now only returns audio/video.

A unit test will confirm

[david-allison]

Good first issue: Produce a unit test which confirms this hypothesis

It should fail on main, and pass before 7a65160

Assuming the test does portray a problem:

Either:

• @Ignore the test, and add a PR (preferred for less experienced developers)
• Fix the issue and check in the fix, along with a passing test

There are many sample tests to follow:

Anki-Android/AnkiDroid/src/androidTest/java/com/ichi2/anki/tests/ContentProviderTest.kt

Lines 975 to 1028 in 06e8ac2

	@Test
	@Synchronized
	fun testQueryCardFromCertainDeck() {
	val deckToTest = testDeckIds[0]
	val deckSelector = "deckID=?"
	val deckArguments = arrayOf(deckToTest.toString())
	val sched = col.sched
	val selectedDeckBeforeTest = col.decks.selected()
	col.decks.select(1) // select Default deck
	val reviewInfoCursor = contentResolver.query(
	FlashCardsContract.ReviewInfo.CONTENT_URI,
	null,
	deckSelector,
	deckArguments,
	null
	)
	assertNotNull(reviewInfoCursor)
	assertEquals("Check that we actually received one card", 1, reviewInfoCursor.count)
	reviewInfoCursor.use {
	it.moveToFirst()
	val cardOrd =
	it.getInt(it.getColumnIndex(FlashCardsContract.ReviewInfo.CARD_ORD))
	val noteID =
	it.getLong(it.getColumnIndex(FlashCardsContract.ReviewInfo.NOTE_ID))
	assertEquals(
	"Check that the selected deck has not changed",
	1,
	col.decks.selected()
	)
	col.decks.select(deckToTest)
	var nextCard: Card? = null
	for (i in 0..9) { // minimizing fails, when sched.reset() randomly chooses between multiple cards
	nextCard = sched.card
	if (nextCard != null && nextCard.nid == noteID && nextCard.ord == cardOrd) break
	try {
	Thread.sleep(500)
	} catch (e: Exception) {
	Timber.e(e)
	} // Reset counts is executed in background.
	}
	assertNotNull("Check that there actually is a next scheduled card", nextCard)
	assertEquals(
	"Check that received card and actual card have same note id",
	nextCard!!.nid,
	noteID
	)
	assertEquals(
	"Check that received card and actual card have same card ord",
	nextCard.ord,
	cardOrd
	)
	}
	col.decks.select(selectedDeckBeforeTest)
	}

[mikehardy]

Oh my, this:

My presumption is that filesInStr was broken in 7a65160

...is correct. It was definitely broken then. And as near as I can tell there is a lot of code and infrastructure around parsing out AV tags (audio, or video) but nothing about images in the upstream anki (where this is all implemented).

A unit test will certainly confirm this

I think the most correct path is likely to lobby for + propose a PR into upstream anki that also implements something like extract_img_tags here next to extract_av_tags https://github.com/ankitects/anki/blob/9ced6be03ec25afef66452a655c88a16bb5eb607/rslib/src/card_rendering/mod.rs#L19 - including fresh parsing of those nodes and the protobuf request/response etc

I think the most expedient path is likely to be re-implementing something like the logic of Media.filesInStr before dropping legacy backend, but focused solely on card contents text parse for images, and adding those to the list already returned by the AV tag backend method: https://github.com/ankidroid/Anki-Android/blame/4d39b5b9dd932d86826dd9dacc5ec2d94603046a/AnkiDroid/src/main/java/com/ichi2/libanki/Media.kt#L218