The reason we should see Black slaves on Portuguese ships and with the Jesuit priests is because to omit them is to deny their role in slavery. Also missing is the Jesuit participation in trading Japanese and Korean slaves. The female "comfort women" I believe were slaves as well, the brief reference to them doesn't make clear it wasn't a voluntary role. Just because Clavell didn't write about them and they didn't appear in 1980 doesn't mean the myth of a slaveless Japan should be perpetuated.
I wasn't trying to suggest that Black people existed in significant numbers in Japan, but that they were present and would have been seen pretty much whenever the Europeans were.