We are currently having issues with our SurveyMonkey orchestration job. It is only pulling back the first 100 responses for each survey. There is reference to the 100 results on the SurveyMonkey documentation (https://api.surveymonkey.net/v3/docs?api_key=3yr7n6m8sjwvm48x8nhxej52#export-the-results-of-a-survey)
Hi @seth.mitchell and thanks for posting!
It sounds like the limit might be defaulting to 100. On the Properties section for the SurveyMonkey component, look for Limit property and the Value of it. If you remove the 100 you should get more. You can find documentation on SurveyMonkey Query here which may also help.
I hope that helps! Let me know how you get on.
Many thanks,
Claire
@ClaireSeniorCommunityManager Thanks for getting back to me! We are still having the issue after removing the limits as @vp120821 also is mentioning.
We are running:
Matillion ETL
Version: 1.65.7 (build 159812d-12014)
AMI Version: 1.65.7
I've successfully queried SurveyMonkey with the latest version of CData's JDBC driver (default options, didn't tweak anything). Used Dbeaver. Received proper number of records, not just one page/100 records.
Latest Cdata jdbc version 22.0.8257
Matillion 1.65.8-1 uses version 21.0.8137.0
@ClaireSeniorCommunityManager could you please share this info with the engineering/development team. It looks like a bug in either SM component or driver version in use.
I've provided more details on case #00055112
FYI @seth.mitchell @vp120821
@SK607582 we too were getting the proper results outside of Matillion and just hitting the SurveyMonkey API directly. We were able to get 1700+ Unique Respondent IDs using Python and iterating over each page. When we hit the Matillion component, we are limited to 100 Unique Respondent IDs.
We're also having this issue. Please help!
Matillion Version: 1.65.8 (build 2caa62f-12676)
I'm not sure, but I do think this bug might have been introduced in a recent version of Matillion, or something. Earlier this year, I think we were able to get results for more than 100 respondent IDs.
Hi all and thank you all so much for posting and for your patience.
The team has added a fix for this in the new 1.66 update. Please take a look and let us know how you get on.
Again, thanks so much for your patience while this was investigated.
Many thanks,
Claire
Hi @seth.mitchell (Customer). I too am facing the same issue. Do update this thread if you are able to find the solution to this.
Hi @ClaireSeniorCommunityManager (Matillion Limited).
I tried to get the row count when the limit is 100 and after removing it. Unfortunately both the counts are same.
Moreover I also tried to increase the pagesize in connection options field of component to 1000, but looks like it is capped at 100 as per the documentation.
So, we need a way to achieve following things:
- Get a list of all the page IDs for {Survey_Title_Responses}
- Iterate all these page IDs for {Survey_Title_Responses} and load it into one table using SurveyMonkey Query component
Hi there @SK607582 and thanks for your post! I've passed that information onto the team.
Many thanks,
Claire
After months of research and frustration we came to the conclusion that the Matillion Component Itself must be either hard coded to limit to 100 responses or the pagination isnt set up correctly. Either possibility means the component is worthless. We were however able to use the API Query Component to get Survey Monkey into Snowflake, a lot more setup is required but in the end, spent less time setting that up than trying to QA the Matillion Component.
Thanks for posting @vp120821. Can you let me know what version of Matillion ETL you are on so that the team can take a further look?
Many thanks,
Claire
@ClaireSeniorCommunityManager (Matillion Limited)
For me it is:
Matillion ETL
Version: 1.61.6 (build a392342-2471)
AMI Version: 1.61.6
Hi @cedric.warren .
Can you please elaborate more on the additional setup that is required to set up API Query component for fetching SurveyMonkey Responses? Thanking you in advance.