Phase I/II Study of Immunotherapy Combination BN-Brachyury Vaccine, M7824, N-803 and Epacadostat (QuEST1)

* INCLUSION CRITERIA: Participants must have histologically or cytologically confirmed any solid tumor (Cohort 1) or castration-resistant prostate cancer (CRPC), Cohorts 2A, 2D and 2R). For the Cohort 1, eligible participants must have a histologically, cytologically or radiographically proven metastatic or locally advanced solid tumor of any type, for which there is no curative standard therapy or standard therapy has failed. Castrate testosterone level (less than 50ng/dl or 1.7nmol/L). (Participants with a malignancy other than prostate cancer are excluded from this criterion). Radiological confirmation of metastatic disease, or Progressive disease at study entry defined as one or more of the following criteria occurring in the setting of castrate levels of testosterone: --Radiographic progression defined as any new or enlarging bone lesions or growing lymph node disease, consistent with prostate cancer OR --prostate-specific antigen (PSA) progression defined by sequence of rising values separated by greater than 1 week (2 separate increasing values over a minimum of 1 ng/ml (Prostate Cancer Clinical Trials Working Group 3 (PCWG3) PSA eligibility criteria). If participants had been on flutamide, PSA progression is documented 4 weeks or more after withdrawal. For participants on bicalutamide or nilutamide disease progression is documented 6 or more weeks after withdrawal. The requirement for a 4-6 week withdrawal period following discontinuation of flutamide, nilutamide or bicalutamide only applies to participants who have been on these drugs for at least the prior 6 months. For all other participants they must stop bicalutamide, nilutamide or flutamide the day prior to enrollment. Asymptomatic or mildly symptomatic form prostate cancer; no use of regularly scheduled opiate analgesics for prostate cancer-related pain. (Participants with a malignancy other than prostate cancer are excluded from this criterion). Participants must agree to continuation of androgen deprivation therapy (ADT) with a gonadotropin-releasing hormone analogue/antagonist or bilateral orchiectomy. (Participants with a malignancy other than prostate cancer are excluded from this criterion). Participants may also continue oral androgen receptor antagonist/anti-androgen therapy (e.g. enzalutamide or abiraterone) unless enrolling to Arm 2.3A or 2.3B due to concerns regarding Cytochrome P450 (CYP)mediated drug-drug interactions epacadostat. Participants must have had the following prior therapy: Testosterone lowering therapy for castration-resistant prostate cancer (CRPC) In addition to continuation of androgen deprivation therapy (ADT) (unless status post bilateral orchiectomy) eligible patients must have received and had PSA or radiographic progression on enzalutamide (or other oral androgen receptor antagonist e.g. darolutamide or apalutamdide) or abiraterone acetate. Participants who have tumors known to be microsatellite instability high/mismatch repair deficient or tumor mutational burden high must have received prior pembrolizumab. Participants with known pathogenic homologous recombination repair mutations for which there is evidence of benefit with poly (adenosine diphosphate-ribose) polymerase (PARP) inhibitors (e.g. BReast CAncer gene 1 (BRCA1). BReast CAncer gene 2 (BRCA2), ataxia-telangiectasia mutated (ATM) must have received prior PARP inhibitor. Age greater than or equal to 18 years. Eastern Cooperative Oncology Group (ECOG) performance status less than or equal to 1 Participants must have adequate organ and marrow function as defined below: * Absolute neutrophil count greater than or equal to 1000/mcL * Platelets greater than or equal to 100,000/mcL * Hemoglobin greater than or equal to 9.0 g/dL * Total bilirubin within normal institutional limits; in participants with Gilbert's, less than or equal to 3.0 mg/dL * Aspartate aminotransferase (AST) serum glutamic-oxaloacetic transaminase (SGOT)/alanine aminotransferase (ALT) serum glutamic-pyruvic transaminase (SGPT) less than or equal to 2.5X upper limit of normal. For subjects with liver involvement in their tumor, AST less than or equal to 3.5. X upper limit of normal (ULN), ALT less than or equal to 3.5 X ULN, and bilirubin less than or equal to 3.0 is acceptable * Creatinine within 1.5X upper limit of normal institutional limits The effects of Bavarian Nordic (BN)-Brachyury, bintrafusp alfa (M7824), Anktiva (N-803), and Epacadostat on the developing human fetus are unknown. For this reason, men and women must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, during the study and maintain such contraception until 4 months following the last dose of any study agent. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her partner s treating physician immediately. Ability of subject to understand and the willingness to sign a written informed consent document. Participants with successfully treated hepatitis C virus (HCV) are eligible if HCV viral load is undetectable. EXCLUSION CRITERIA: Participants who are immunocompromised as follows: * Human immunodeficiency virus positivity due to the potential for decreased tolerance, and potential to be at risk for severe side effects with immunotherapies. These concerns are relevant to all drugs, as drug-drug interactions among antiretrovirals and immunotherapies are yet uncharacterized. * Chronic administration (defined as daily or every other day for continued use greater than 14 days) of systemic corticosteroids or other immune suppressive drugs, within 28 days before treatment on study. Nasal, or inhaled steroid, topical steroid creams and eye drops for small body areas are allowed. Physiologic doses of steroids are permitted, e.g., a participant taking hydrocortisone for adrenal insufficiency or a patient recently on abiraterone (administered with 10 mg of prednisone daily) who is tapering off of prednisone is allowed to continue that taper. * Participants who have undergone allogeneic peripheral stem cell transplantation, or solid organ transplantation requiring immunosuppression * Active autoimmune disease, except participants with type 1 diabetes mellitus, vitiligo, psoriasis, hypo- or hyperthyroid disease not requiring current immunosuppression, or with other endocrine disorders on replacement hormones or are not excluded if the condition is well controlled. Prostate cancer participants with a history of brain/leptomeningeal metastasis, since these participants have a very poor prognosis, and immunotherapy may take time to lead to beneficial clinical effects. Participants with brain or CNS metastases enrolling to arm 1.1 are eligible if they are status post definitive radiotherapy or surgery, and are asymptomatic History of allergic reactions attributed to compounds of similar chemical or biologic composition to study agents to be used in the cohort the subject will be enrolled into. Known allergy to eggs, egg products, aminoglycoside antibiotics (for example, gentamicin or tobramycin). Any condition which, in the opinion of the investigator, would prevent full participation in this trial (including the long-term follow-up), or would interfere with the evaluation of the trial endpoints. Participants with prior investigational drug, chemotherapy, immunotherapy or any prior radiotherapy (except for palliative bone directed therapy) within the past 28 days prior to enrollment, except if the investigator has assessed that all residual treatment-related toxicities have resolved or are minimal and feel the participant is otherwise suitable for enrollment. Uncontrolled intercurrent acute or chronic illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure (\>New York Heart Association Class I), hepatic disease, unstable angina pectoris, serious cardiac arrhythmia, requiring medication, uncontrolled hypertension (Systolic blood pressure (SBP) \>170/ diastolic blood pressure (DBP)\>105) or psychiatric illness/social situations within 12 months that would limit compliance with study requirements. Use of herbal products that may decrease PSA levels (e.g., saw palmetto) Participants who have had cytotoxic chemotherapy for metastatic castration-resistant prostate cancer within the past 3 months. (Participants who have had docetaxel for metastatic castration sensitive per Chemohormonal Therapy Versus Androgen Ablation Randomized Trial for Extensive Disease in Prostate Cancer (CHAARTED) data may enroll as long as they did not have progressive disease while on docetaxel and are 3 months removed from treatment, with all treatment related toxicities resolving to at least grade 1.) Participants who have undergone major surgery within 4 weeks of enrollment. A biopsy will not preclude a participant from starting study. Participants with a history of hepatitis B (HBV) are excluded due to potential risk for viral reactivation and resulting liver injury in persons with latent HBV Subjects unwilling to accept blood products or blood transfusions as medically indicated. As there is a risk of severe bleeding with M7824, participants must be willing to receive blood transfusions if medically necessary for their own safety For participants enrolling in Arm 2.3A and for participants who may be randomized to Arm 2.3B, the following additional exclusion criteria will apply: * Subjects receiving Monoamine Oxidase Inhibitors (MAOIs) or a drug which has significant MAOI activity (meperidine, linezolid, methylene blue) within the 21 days before initiation of study therapy are excluded. * Since epacadostats metabolism may be altered by drugs that inhibit uridine diphosphate (UDP)-glucuronosyltransferase, UGT1A9 (see Appendix D), Participants receiving such drugs within 21 days of initiation of study therapy are excluded. * Participants receiving agents that are substrates of Cytochrome P450 1A2 (CYP1A2), cytochrome P450 2C8 (CYP2C8), and cytochrome P450 2C19 (CYP2C19) or affected by organic anion transporting polypeptide 1B1 (OATP1B1) or organic anion transporting polypeptide 1B3 (OATP1B3) within 21 days of initiation of study therapy or 5 half-lives of the agent (whichever is shorter) are excluded. Participants receiving medications that are substrates of these enzymes/transporters but are not listed in the appendix or participants receiving substrates of CYP2B6 and CYP3A will be evaluated on a case-by-case basis prior to enrollment. Participants who consume caffeine are eligible but must agree to moderate consumption. * Subjects receiving coumarin-based anticoagulants (e. Coumadin) are excluded. * Subjects having any history of Serotonin Syndrome (SS) after receiving serotonergic drugs are excluded. --Participants with a QTc corrected for heart rate by Fridericia's formula (QTcF) interval \> 480 milliseconds at the screening are excluded. In the event that a single QTcF is \> 480 milliseconds, the subject may enroll if the average QTcF for the 3 electrocardiogram (ECG's) is \< 480 milliseconds. For subjects with an intraventricular conduction delay (QRS interval \> 120 milliseconds), the corrected JT interval, (JTc) interval may be used in place of the QTc. The JTc must be \< 340 milliseconds if JTc is used in place of the QTc. QTc prolongation due to pacemaker may enroll if the JTc is normal. * Subjects with left bundle branch block are excluded. * Pregnant women are excluded from this study because investigational agents used in this study (BN-Brachyury, M7824, N-803, and/or Epacadostat) could have teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with these investigational agents, breastfeeding should be discontinued if the mother is treated with either of them.