Fix for 400/502 error "bad request" due to large headers and too many cookies

Hi,

Just in case others are hitting the same problem. You might be noticing that some users are randomly having issues logging in, form submissions etc. We noticed this and traced it back to 400 errors related to larger request headers.
It seems that the switch to GA4 and the increasing use of Hotjar are the cause of the problem. Both of these services set cookies by default at the nsw.gov.au domain and both set a unique cookie per property. The result is that if a users visits a lot of different government sites they can end up with over 8k of cookies (+headers) being sent to all nsw.gov.au websites. Most web application servers and reverse proxys tend to have an 8k limit by default.

We’ve increased the limits on our NSW DDS hosting service https://digitalnsw.pretagov.com.au/ and also implemented measures to try to cut out some of the unwanted cookies.

However it would be good practice if other services do what we have also done and ensure that GA etc use a more specific domain. e.g. gtag('config', 'PROP_ID', {'cookie_domain': 'subdomain.nsw.gov.au'})

1 Like

Hi @djay , our sincere apologies for the delay in response to this.

We don’t have this issue as we do not use hotjar and our GA4 is installed using google tag manager which only fires if you are not logged in to the CMS.

This isn’t an issue because one site uses hotjar or GA4 or tag manager. This is an issue because many nsw.gov.au subsites do and the by default all the cookies are set to the base domain of nsw.gov.au. If a user has visited a number of those sites then the cookies start building up.

Of course, @djay I understand the issue now. The accumulation of cookies across multiple nsw.gov.au subsites is caused by the implementation (which is domain specific) on the oneCX nsw.gov.au site. While individual sites might not trigger the issue on their own, the cumulative effect on users navigating various government sites is what leads to the challenge.

Thanks again for posting about the issue.

1 Like

@Digital.NSW exactly. Hopefully anyone else doing custom services (not oneCX) that encounters this problem will find this.
and everyone should customise their tags to only be set for their subdomain only.

I believe getting nsw.gov.au back on the PSL would eliminate the issue of HotJar and Google Analytics sharing cookies across domains from unrelated departments too.

Is anyone in a position to confirm that and potentially run it up to the right people to sign off on?

PSL: https://publicsuffix.org/
Where it was removed: 547985 - Comment out nsw.gov.au in PSL