check upsample max times to avoid invalid frame buffer access

check upsample max times to avoid invalid frame buffer access #553