# How to load data from GitHub Raw?

**URL:** <https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428>\
**Category:** Matillion ETL\
**Created:** [September 2, 2021, 2:11pm UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428 "2021-09-02T14:11:24Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 2, 2021, 2:11pm UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/1 "2021-09-02T14:11:24Z")

</div>

Hi all,

I try to load json files that are in a private GitHub Repo.

The issue is that the API Query component can't handle it, as it is "text/plain" (even tough it's perfectly fine json)

Alternative would be to use the Data Transfer component, but that can't handle the Bearer Token Auth, which is needed for the private Repo.

Do I miss something or you guys have other ideas how to load these files?

---

<div class="post-metadata">

**Author:** ![Bryan](https://avatars.discourse-cdn.com/v4/letter/b/db5fbb/32.png) [@Bryan](https://matillioncommunity.discourse.group/u/Bryan)\
**Post date:** [September 2, 2021, 9:00pm UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/2 "2021-09-02T21:00:15Z")

</div>

Hi [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)​ , just to make sure I am understand what you are attending to retrieve... Are you retrieving files from a repo in GitHub or pulling some other GitHub information?

---

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 3, 2021, 6:43am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/3 "2021-09-03T06:43:01Z")

</div>

The file content itself, as you would get it via the Raw endpoint:

---

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 3, 2021, 6:43am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/4 "2021-09-03T06:43:39Z")

</div>

![0694G00000GG7GYQA1_0D74G000007jeVqSAI](https://us1.discourse-cdn.com/flex002/uploads/matillioncommunity/original/1X/41f8d1b5b4c5be28a9464172d9394e3d3c938aab.png)

---

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 6, 2021, 11:21am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/5 "2021-09-06T11:21:17Z")

</div>

Thanks for your proposal. I checked it, but it didn't work. As far as I understood, that's the header that Matillion is sending to GitHub, right?

So Matillion to Github communication is not an issue here, that works.

The problem is the response, Github to Matillion, which gives this error:

_Error: Invalid JSON markup. Expected json, but instead found [text/plain; charset=utf-8]._

&nbsp;

However, the content is proper JSON (the call from Github to Matillion is "text/plain"). Content of the reponse (This works in Postman, but it detects it as text, not as JSON):

_{_

_&nbsp;&nbsp;&nbsp;&nbsp;"type":&nbsp;"record",_

_&nbsp;&nbsp;&nbsp;&nbsp;"name":&nbsp;"vacancy",_

_&nbsp;&nbsp;&nbsp;&nbsp;"namespace":&nbsp;"marketplace.core.entity",_

_&nbsp;&nbsp;&nbsp;&nbsp;"fields":&nbsp;[_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"name":&nbsp;"id",_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"type":&nbsp;"string",_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"doc":&nbsp;"Unique&nbsp;id&nbsp;of&nbsp;the&nbsp;vacancy&nbsp;(UUIDv4)"_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;},_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"name":&nbsp;"accountId",_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"type":&nbsp;"string",_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"doc":&nbsp;"Id&nbsp;of&nbsp;the&nbsp;account&nbsp;to&nbsp;which&nbsp;owns&nbsp;the&nbsp;vacancy"_

_&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;},_

_..._

_&nbsp;&nbsp;&nbsp;&nbsp;]_

_}_

---

<div class="post-metadata">

**Author:** ![Bryan](https://avatars.discourse-cdn.com/v4/letter/b/db5fbb/32.png) [@Bryan](https://matillioncommunity.discourse.group/u/Bryan)\
**Post date:** [September 3, 2021, 3:24pm UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/6 "2021-09-03T15:24:14Z")

</div>

Thanks for providing the info and screenshot. That is really helpful. One thing that comes to mind is if you are using the GitHub API to pull file contents you would typically need to include a specific "Accept" attribute and value in the header. In the API Query component make sure you have this connection option configured:

 ![0694G00000GGBAeQAP_0D74G000007jeo9SAA](https://us1.discourse-cdn.com/flex002/uploads/matillioncommunity/original/1X/dcf33e05bb51a888ab9f59ea71f8d48880863f2b.jpeg)

&nbsp;

If this doesn’t work perhaps you could you post the API Profile you are using to get the file contents from GitHub and the API Query configuration you currently have? Feel free to obfuscate any content you deem intellectual property or security related to protect your GitHub environment and company. I hope the above helps.

---

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 6, 2021, 11:21am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/7 "2021-09-06T11:21:45Z")

</div>

the used rsd file

---

<div class="post-metadata">

**Author:** ![ian.funnell](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/ian.funnell/32/605_2.png) [@ian.funnell](https://matillioncommunity.discourse.group/u/ian.funnell)\
**Post date:** [September 7, 2021, 7:32am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/8 "2021-09-07T07:32:16Z")

</div>

Hi [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)​,

What version of Matillion ETL are you using there? It worked OK for me on a Version: 1.55 (for Snowflake) querying [this raw JSON file](https://raw.githubusercontent.com/MatillionDeveloper/Orchestration/main/XLS/Snowflake/METL-1.54/XLS_Extract_Load-SFAWS-1.54.json) from the Developer Relations account...

 ![0694G00000GGOtNQAX_0D74G000007jfgaSAA](https://us1.discourse-cdn.com/flex002/uploads/matillioncommunity/original/1X/5cd1267fba8a794f42301a215928709957e143ef.png)

Best regards,

Ian

---

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 7, 2021, 7:41am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/9 "2021-09-07T07:41:30Z")

</div>

We're using Version 1.54.10&nbsp;for AWS Redshift.

---

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 7, 2021, 7:54am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/10 "2021-09-07T07:54:09Z")

</div>

Ok, got it running when recreating the profile with the wizard. Will check what may have caused it

---

<div class="post-metadata">

**Author:** ![Manuel](https://sea2.discourse-cdn.com/flex002/user_avatar/matillioncommunity.discourse.group/manuel/32/940_2.png) [@Manuel](https://matillioncommunity.discourse.group/u/Manuel)\
**Post date:** [September 7, 2021, 7:56am UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/11 "2021-09-07T07:56:25Z")

</div>

Ok, looks like

&nbsp;

&nbsp;_\<api:set attr="BackwardsCompatibilityMode" value="true" /\>_

&nbsp;

is needed here. Without this being set, you'll run into the error described above.

---

<div class="post-metadata">

**Author:** ![Bryan](https://avatars.discourse-cdn.com/v4/letter/b/db5fbb/32.png) [@Bryan](https://matillioncommunity.discourse.group/u/Bryan)\
**Post date:** [September 7, 2021, 2:56pm UTC](https://matillioncommunity.discourse.group/t/how-to-load-data-from-github-raw/1428/12 "2021-09-07T14:56:42Z")

</div>

I should have known that! I have ran into issues with the BackwardsCompatibilityMode being true when it needed to be false and vice-versa. The errors I have seen where different than the one you are getting which probably makes sense because you are going after a completely different API than I was. Glad you got it working!
